Skip to content

Add: report NEXT_LEVEL reservation stalls - #1613

Merged
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
puddingfjz:feat/report-group-reservation-stalls
Aug 3, 2026
Merged

Add: report NEXT_LEVEL reservation stalls#1613
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
puddingfjz:feat/report-group-reservation-stalls

Conversation

@puddingfjz

@puddingfjz puddingfjz commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • detect a blocked NEXT_LEVEL group reservation when an idle reserved target has queued single work
  • emit one native warning after the structural condition persists for five seconds
  • include the group slot, busy targets, idle-but-queued targets, and their single FIFO head slots

Behavior

This is diagnostic only. It does not classify the condition as a deadlock, release the reservation, or otherwise change scheduling policy. Reporting is edge-triggered per group/stall episode and uses a native no-throw sink rather than a Python logger callback.

Warning sink

The sink is noexcept and runs on the scheduler dispatch path, so it formats into automatic storage and emits with a single write(2). It allocates nothing — a throwing allocation inside a noexcept sink would call std::terminate, precisely under the resource pressure worth diagnosing — takes no stdio lock that a forked Worker child could inherit held, and leaves nothing running for process exit to race. A message that does not fit the buffer loses its tail and keeps its newline.

reservation_stall_episode_ is confined to sched_thread_: update_reservation_stall() writes it under loop_mu_ and reservation_stall_deadline() reads it under completion_mu_, which is race-free only because one thread does both.

Dependency

#1612 is merged and this branch is rebased onto it. The scheduler's timed wait (wait_until, armed only while a stall episode is open and unreported) is introduced here, not by #1612.

Testing

  • C++ UT: 74/74 ctest targets, test_scheduler 60/60
  • Python UT: pytest tests/ut -m "not requires_hardware" — 1026 passed, 7 skipped
  • warning-sink formatting and truncation checked standalone under ASan: nominal input, empty target lists, 4096 targets truncated into a 512-byte buffer, and every buffer capacity from 2 to 600 (len == strlen(buf) and len <= cap - 1 throughout)
  • pre-commit clean (clang-format, clang-tidy, cpplint, markdownlint)

@puddingfjz
puddingfjz force-pushed the feat/report-group-reservation-stalls branch from d688ef9 to 1a3a55c Compare July 31, 2026 09:41
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e4dcfba6-9d2c-47fc-910a-b8e0513369a9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The scheduler now uses wake generations and completion events for waiting. NEXT_LEVEL dispatch returns reservation details, tracks structural stalls, and emits timed diagnostics. Queue inspection and tests cover blocked groups, queued singles, duplicate completions, and wake-up behavior.

Changes

Scheduler wake and reservation stall handling

Layer / File(s) Summary
Scheduler and queue contracts
src/common/hierarchical/scheduler.h, src/common/hierarchical/types.*
Scheduler configuration and dispatch results now include reservation-stall data. NextLevelReadyQueues supports worker-specific front inspection.
Wake-generation scheduler waiting
src/common/hierarchical/scheduler.cpp, src/common/hierarchical/scheduler.h, docs/scheduler.md
Scheduler notifications advance a wake generation. The scheduler waits on unconsumed generations, terminal completions, or reservation-stall deadlines.
Reservation-aware dispatch and diagnostics
src/common/hierarchical/scheduler.cpp, src/common/hierarchical/worker.cpp, docs/scheduler.md
Group dispatch reports busy targets and queued target heads. Stall episodes produce one diagnostic after the configured interval.
Scheduler behavior validation
tests/ut/cpp/hierarchical/test_scheduler.cpp
Tests cover stall diagnostics, blocked-group waiting, incremental dispatch, duplicate completion wake-ups, and invalid completion cleanup.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Scheduler
  participant ReadyQueues
  participant CompletionFIFO
  participant WorkerDiagnosticSink
  Scheduler->>ReadyQueues: inspect target queue heads
  ReadyQueues-->>Scheduler: reservation and queue state
  Scheduler->>Scheduler: track wake generation and stall deadline
  CompletionFIFO-->>Scheduler: terminal completion
  Scheduler->>WorkerDiagnosticSink: report reservation stall
Loading

Possibly related PRs

Poem

A rabbit watched the wake count grow,
While queued tasks waited in a row.
Stall heads spoke after five long beats,
Busy paws freed waiting seats.
The scheduler hopped, alert and bright.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: reporting NEXT_LEVEL reservation stalls.
Description check ✅ Passed The description directly explains stall detection, warning behavior, diagnostic contents, and testing for the changeset.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@puddingfjz puddingfjz changed the title Feat/report group reservation stalls Add: report NEXT_LEVEL reservation stalls Jul 31, 2026
@puddingfjz
puddingfjz force-pushed the feat/report-group-reservation-stalls branch from 1a3a55c to 3b3ebb1 Compare July 31, 2026 15:57

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (4)
tests/ut/cpp/hierarchical/test_scheduler.cpp (2)

1179-1186: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value

Assert the worker pointers before dereferencing them.

manager.get_worker_by_id returns nullptr for an unregistered id. Scheduler::dispatch_next_level_group in src/common/hierarchical/scheduler.cpp checks for that case explicitly. Worker ids 0 and 1 are registered in SetUp(), so the current test is safe. Add ASSERT_NE(..., nullptr) so a future registration change fails as an assertion instead of a segmentation fault inside a lock scope.

🛡️ Proposed change
         WorkerThread *manager_worker_a = manager.get_worker_by_id(WorkerType::NEXT_LEVEL, 0);
         WorkerThread *manager_worker_b = manager.get_worker_by_id(WorkerType::NEXT_LEVEL, 1);
+        ASSERT_NE(manager_worker_a, nullptr);
+        ASSERT_NE(manager_worker_b, nullptr);
         while ((!manager_worker_a->idle() || !manager_worker_b->idle()) &&
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ut/cpp/hierarchical/test_scheduler.cpp` around lines 1179 - 1186, Add
ASSERT_NE checks for manager_worker_a and manager_worker_b immediately after
their get_worker_by_id calls and before the idle() polling loop, so null worker
registrations fail the test safely before dereferencing either pointer.

786-796: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Gate the diagnostic field checks with an assertion.

If the report does not arrive within 200 ms, line 790 fails and the test continues. Lines 791-796 then compare default-initialized capture fields and emit five more failures. Use ASSERT_EQ for the report-count check so the test stops at the root cause.

The same pattern applies to the exact dispatched_count() checks at lines 797, 810-811, and 820-821. Those are less severe because a stale count fails only one expectation.

♻️ Proposed change
-    EXPECT_EQ(stall_capture.report_count.load(std::memory_order_acquire), 1);
+    ASSERT_EQ(stall_capture.report_count.load(std::memory_order_acquire), 1);
     EXPECT_EQ(stall_capture.group_slot, group.task_slot);
-    EXPECT_EQ(stall_capture.busy_target_count, 1u);
+    ASSERT_EQ(stall_capture.busy_target_count, 1u);
     EXPECT_EQ(stall_capture.busy_target_worker_ids[0], 1);
-    EXPECT_EQ(stall_capture.idle_queued_target_count, 1u);
+    ASSERT_EQ(stall_capture.idle_queued_target_count, 1u);
     EXPECT_EQ(stall_capture.idle_queued_target_worker_ids[0], 0);
     EXPECT_EQ(stall_capture.idle_queued_single_head_slots[0], single_a.task_slot);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ut/cpp/hierarchical/test_scheduler.cpp` around lines 786 - 796, Use
ASSERT_EQ for the stall_capture.report_count check before validating the
diagnostic fields, so the test returns immediately when no report arrives. Apply
the same assertion-strengthening to the exact dispatched_count() checks at the
indicated points, preserving their existing expected values and surrounding test
logic.
src/common/hierarchical/scheduler.h (2)

108-146: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Document the single-writer-thread invariant on reservation_stall_episode_.

reservation_stall_deadline() reads reservation_stall_episode_ under completion_mu_ (in run()), and update_reservation_stall() writes it under loop_mu_ (in dispatch_ready()). This is safe only because both calls execute exclusively on the scheduler thread. Add a short comment near the member to record this invariant, so a future change that invokes either method from another thread does not introduce a data race.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/hierarchical/scheduler.h` around lines 108 - 146, Add a short
comment immediately above reservation_stall_episode_ documenting that it is
accessed only by the scheduler thread, including its reads in
reservation_stall_deadline() and writes in update_reservation_stall(). Do not
change the locking or surrounding dispatch logic.

59-69: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Document the pointer-lifetime contract on ReservationStallDiagnostic.

busy_target_worker_ids, idle_queued_target_worker_ids, and idle_queued_single_head_slots are raw pointers. In scheduler.cpp, update_reservation_stall builds this struct from dispatch_result.busy_target_worker_ids.data() and similar calls, where dispatch_result is a temporary owned by the caller of dispatch_ready(). The sink call is synchronous, so the pointers stay valid only for the duration of that call.

Add a comment on the struct that states this constraint. A future sink implementation that stores these pointers past the callback invocation reads freed memory.

📝 Proposed documentation addition
     struct ReservationStallDiagnostic {
+        // All pointer/count fields below reference memory owned by the caller
+        // of the sink and are valid only for the duration of the sink call.
+        // Do not store these pointers past the callback invocation.
         TaskSlot group_slot{INVALID_SLOT};
         const int32_t *busy_target_worker_ids{nullptr};
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/hierarchical/scheduler.h` around lines 59 - 69, Add a
documentation comment to the ReservationStallDiagnostic struct stating that its
raw pointer fields are valid only during the synchronous ReservationStallSink
callback and must not be retained afterward, since they reference caller-owned
temporary storage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/common/hierarchical/worker.cpp`:
- Around line 37-54: Update report_reservation_stall so the synchronously
invoked diagnostic sink never performs potentially blocking stderr operations.
Replace flockfile and repeated std::fprintf calls with an existing bounded
non-blocking native sink, or add explicit EPIPE and backpressure handling that
prevents SIGPIPE and unbounded blocking before this sink is registered.

In `@tests/ut/cpp/hierarchical/test_scheduler.cpp`:
- Around line 1189-1208: Move the EXPECT_EQ assertions for worker_a and worker_b
dispatch counts until after the optional notify_ready retry completes. Keep the
initial wait and retry as best-effort cleanup protection, then assert the final
counts before conditionally calling complete() so a successful retry does not
leave a prior test failure.

---

Nitpick comments:
In `@src/common/hierarchical/scheduler.h`:
- Around line 108-146: Add a short comment immediately above
reservation_stall_episode_ documenting that it is accessed only by the scheduler
thread, including its reads in reservation_stall_deadline() and writes in
update_reservation_stall(). Do not change the locking or surrounding dispatch
logic.
- Around line 59-69: Add a documentation comment to the
ReservationStallDiagnostic struct stating that its raw pointer fields are valid
only during the synchronous ReservationStallSink callback and must not be
retained afterward, since they reference caller-owned temporary storage.

In `@tests/ut/cpp/hierarchical/test_scheduler.cpp`:
- Around line 1179-1186: Add ASSERT_NE checks for manager_worker_a and
manager_worker_b immediately after their get_worker_by_id calls and before the
idle() polling loop, so null worker registrations fail the test safely before
dereferencing either pointer.
- Around line 786-796: Use ASSERT_EQ for the stall_capture.report_count check
before validating the diagnostic fields, so the test returns immediately when no
report arrives. Apply the same assertion-strengthening to the exact
dispatched_count() checks at the indicated points, preserving their existing
expected values and surrounding test logic.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c6f392a1-58b3-4829-a64a-9d5720437d0d

📥 Commits

Reviewing files that changed from the base of the PR and between f2a20ca and 3b3ebb1.

📒 Files selected for processing (7)
  • docs/scheduler.md
  • src/common/hierarchical/scheduler.cpp
  • src/common/hierarchical/scheduler.h
  • src/common/hierarchical/types.cpp
  • src/common/hierarchical/types.h
  • src/common/hierarchical/worker.cpp
  • tests/ut/cpp/hierarchical/test_scheduler.cpp

Comment thread src/common/hierarchical/worker.cpp
Comment thread tests/ut/cpp/hierarchical/test_scheduler.cpp
@ChaoWao
ChaoWao force-pushed the feat/report-group-reservation-stalls branch from 3b3ebb1 to fea8387 Compare August 3, 2026 01:48
@ChaoWao

ChaoWao commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Rebased onto the reconciled #1612 head (wake model merged with #1541's run-partitioned queues) and addressed review feedback:

Rebase

  • Stacked onto Fix: block scheduler while group heads wait as reconciled with Update: make whole-run FIFO admission failure-safe #1541. The stall peek is now partition-aware (try_front_single(worker_id, run_id, slot)), consistent with the run-snapshot dispatch model; the unpartitioned overload remains for the no-active_run_cb path.

Review items

  1. CodeRabbit (Major) — sink on the blocking dispatch path: fixed. The production sink now assembles the message into a std::string and writes it on a detached std::thread; a slow or full stderr pipe can no longer wedge the scheduler. The Config field documents the no-block contract.
  2. CodeRabbit (Major) — retry block cannot recover the outcome: comment clarified. The block's purpose is teardown unblocking, not outcome recovery; wording updated so the intent is unambiguous.

Behavior

Unchanged from the original design: structural discriminator (idle reserved target with queued single work), 5 s default threshold, one report per episode, no self-healing, no scheduling-policy change.

Testing

  • Local: 41 tests × 50 iterations clean (includes ReportsStructuralStallOncePerEpisode); full tests/ut/cpp build clean.
  • CI will re-run on this push.

@ChaoWao
ChaoWao force-pushed the feat/report-group-reservation-stalls branch from fea8387 to 5f01440 Compare August 3, 2026 02:14
@ChaoWao

ChaoWao commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Rebased onto the updated #1612 head (acfa5d8e, the #1583-reconciled wake model). No semantic changes to the watchdog; the same 7-file delta applies on the new base. Local: 57 tests (incl. ReportsStructuralStallOncePerEpisode) x 50 iterations clean; clang-format clean. CI re-running on this push.

@ChaoWao
ChaoWao force-pushed the feat/report-group-reservation-stalls branch 2 times, most recently from 6c9e21e to 1c9276f Compare August 3, 2026 03:37
A blocked NEXT_LEVEL group head reserves every target against single
dispatch, so a target that goes idle while a sibling still runs sits on its
own queued singles until the whole group launches. That is the designed
all-or-nothing reservation and not a fault, but nothing distinguished it
from a genuine stall, and it is invisible from outside the scheduler.

Detect the structural shape — a blocked head with at least one reserved
target that is idle and has a non-empty single FIFO — and report it once per
episode after it persists for five seconds, carrying the group slot, busy
target IDs, idle-but-queued target IDs, and their FIFO head slots. A head
change or the condition clearing starts a new episode. This is diagnostic
only: it does not classify the state as a deadlock, release the reservation,
or change placement. The scheduler arms a wait_until deadline only while an
episode is open and unreported, so a parked scheduler still parks.

The sink is noexcept and runs on the dispatch path, so it formats into
automatic storage and emits with one write(2). It allocates nothing — a
throwing allocation there would call std::terminate under exactly the
resource pressure worth diagnosing — takes no stdio lock a forked Worker
child could inherit held, and leaves nothing running for process exit to
race. A message that does not fit loses its tail and keeps its newline.

reservation_stall_episode_ is confined to sched_thread_: update_reservation_
stall() writes it under loop_mu_ and reservation_stall_deadline() reads it
under completion_mu_, which is race-free only because one thread does both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ChaoWao
ChaoWao force-pushed the feat/report-group-reservation-stalls branch from 1c9276f to a85d07d Compare August 3, 2026 03:43
@ChaoWao
ChaoWao merged commit 810fbcd into hw-native-sys:main Aug 3, 2026
46 of 48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants